SynapseML Development Setup
- Install JDK 11
- You may need an Oracle login to download.
- Install SBT
- Fork the repository on GitHub
- See how to here: Fork a repo - GitHub Docs
- Clone your fork
git clone https://github.com/<your GitHub handle>/SynapseML.git
- This command will automatically add your fork as the default remote, called
origin
- Add another Git Remote to track the original SynapseML repo. It's recommended to call it
upstream
:git remote add upstream https://github.com/microsoft/SynapseML.git
- See more about Git remotes here: Git - Working with remotes
- Go to the directory where you cloned the repo (for instance,
SynapseML
) withcd SynapseML
- Run sbt to compile and grab datasets
sbt setup
- Install IntelliJ
- Configure IntelliJ
- Install Scala plugin during initialization
- OPEN the SynapseML directory from IntelliJ
- If the project doesn't automatically import, click on
build.sbt
and import the project
- Prepare your Python Environment
- Install Miniconda
- Note: if you want to run conda commands from IntelliJ, you may need to select the option to add conda to PATH during installation.
- Activate the
synapseml
conda environment by runningconda env create -f environment.yml
from thesynapseml
directory.noteIf you're using a Windows machine, remove
horovod
requirement in the environment.yml file, because horovod installation only supports Linux or macOS. Horovod is used only for namespacesynapse.ml.dl
.
- Update the ScalaTest Configuration Template
- In IntelliJ, select the sandwich menu in the top left.
- Select Run, then select Edit Configurations. At the bottom of the pop-up, select Edit Configuration Templates.
- Select ScalaTest from the list on the right
- Under VM options, add
--add-exports java.base/sun.nio.ch=ALL-UNNAMED
. Apply the changes.
NOTE
If you will be regularly contributing to the SynapseML repo, you'll want to keep your fork synced with the upstream repository. Please read this GitHub doc to know more and learn techniques about how to do it.
Publishing and Using Build Secrets
To use secrets in the build, you must be part of the synapsemlkeyvault
and Azure subscription. If you're MSFT internal and would like to be
added, reach out to synapseml-support@microsoft.com
SBT Command Guide
Scala build commands
compile
, test:compile
and it:compile
Compiles the main, test, and integration test classes respectively
test
Runs all synapsemltests
scalastyle
Runs scalastyle check on main
test:scalastyle
Runs scalastyle check on test
unidoc
Generates documentation for scala sources
Python Commands
createCondaEnv
Creates a conda environment synapseml
from environment.yml
if it doesn't already exist.
This env is used for python testing.
Activate this env before using python build commands.
cleanCondaEnv
Removes synapseml
conda env
packagePython
Compiles scala, runs python generation scripts, and creates a wheel
generatePythonDoc
Generates documentation for generated python code
installPipPackage
Installs generated python wheel into existing env
testPython
Generates and runs python tests
Environment + Publishing Commands
getDatasets
Downloads all datasets used in tests to target folder
setup
Combination of compile
, test:compile
, it:compile
, getDatasets
package
Packages the library into a jar
publishBlob
Publishes Jar to SynapseML's Azure blob-based Maven repo. (Requires Keys)
publishLocal
Publishes library to the local Maven repo
publishDocs
Publishes scala and python doc to SynapseML's Azure storage account. (Requires Keys)
publishSigned
Publishes the library to Sonatype staging repo
sonatypeRelease
Promotes the published Sonatype artifact